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ABSTRACT 

The purposes of th’s project were to develop * 
profile of the individual physician's practice, test the physician in 
the major areas of his practice, and provide educational consultation 
according to practice profile and test results. A test Lank of 1,800 
5-option multiple choice questions was classified into 18 categories 
based on classi f ica t ion of diseases, with three levels or 
sophistication represented in each category. Questions from about 
five categories were randomly selected for each of 37 participating 
physicians. Each physician's categories were determined from his 
practice profile, whi^h was determined in a week of observation by a 
medical secretary. The resulting data were used by educational 
consultants, who net with the individual physicians to plan 
educational programs to meet their needs. The project fund that the 
procedure holds potential as an aid in educational planning by highly 
motivated physicians, but cautions that it is too narrow tc be useful 
in evaluating physician performance. Also, the? test bank, although 
useful in principle, will require modification before it will succeed 
in practice. <BH) 
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1, Proposal 



A major problem facing those responsible for the continuing 
education of puysicians is identification of education.il needs, so 
that program planning can be responsive to the requirements ol prac- 
ticing physicians. Consideration of this problem at the University 
of Wisconsin led to the thesis that medical practices vary greatly, 
and consequently educational needs must be identified in terms of 
the individual practitioner. 

To explore this concept, a research project was ricsigncJ to 

1. Gather data to develop a profile of a physician's 
prnc t ice . 

2. Test the physician in the major areas of his 
practice . 

3. Frovide educational consu 1 tat ion relevant to his 
practice profile and test results. 

The research was conducted under contract no, N1H 70-4003 
with the Continuing Cduca.ion Branch, Division of Physician Manpower, 
bureau of health Professions education and Manpower Training, 

National Institutes of Health. 

i II. Methodology 

% 

C 

^ Obtaining Pa c t lc l pants 
C 

\ In April of 1968 the principal investigator presented thr 

goals ami procedures of the study at a series of regional continuing 
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education inert i i\yx in Wisconsin and invited, physicians to take part. 

In addition, a number of telephone contacts were made with physicians 
who had aeon generally .supportive of departmental programs in the past. 
Although difficulty in recruiting participants was anticipated, little 
persuasion was required and a number of participants were volunteers 
who had learned of the study from colleagues. 

As a result, 37 private practitioners took part in the 
project. The distribution was 28 in general practice, four in internal 
medicine* four pediatrics and one surgery. University Child Health 
Service (UCHS) asked to he incLuded and two scaff pediatricians parti- 
cipated. Of the private prac t i t ionr rs , 36 were from Wisconsin and one 
from Iowa, Data on the 37 participants is presented in lable 1. 
(Individuals will be represented by code numbers in all data presen- 
tations to preserve anonymity) 

Dove lopment of Tcs * Rank 

In anticipation of the study, collection of test questions 
from a variety of sources began months in advance. Selection was 
restricted primarily to five-option multiple choice items, although 
some variation in the number of options was allowed. A test bank 
of approximately 1,800 items was developed for the i *y . 

The items were then classified ir. 13 categories (see 
Table 2). Seventeen categories were based generally on the Inter- 
national Classification of Diseases, Adapted. Category 18 was 
included for those physicians who saw a large number of patients with 
hypertension within Category 6, Diseases of the Circulatory System. 
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Kneli item was then assigned a level ot sophist if at i on . depend i .i> 
mi whether the i n fc> rmat. i on presented in the question pertained to: 

Level \ - n common clinical situation and "on the spot 1 
dec i s i on . 

Level 1 - a decision requiring commonly available 
di acrostic tools and procedures. 

Level 3-a problem or technique requiring special 
knowledge or training. 

In cadi category two questions were selected at each level, 
so then? would he a total of si* standard questions to be asked of a 
participant who qualified for testing in that category. 

The test items were converted to an ALGOL format usable 
by the Hu r roughs B.S500 computer. 

Coll ect io n of Practice D ata 

In order to obtain data on the physician's practice a medical 
secretary was sent to his office Mr a period of one week. During this 
time she recorded nine items of information on each patient contact 
,3ee Table 3 for types of information recorded). Data was normally 
collected from Monday through Thursday, resulting in three or four days' 
data depending on whether the physician had a day off during that 
period. Ail patient contacts were recorded, whether they occurred in 
the office hospital, hor.c, or over the telephone. 

Kro,< this data recording form, it was possible to determine 
certain ciiarnc ler is l ics of a physician's practice. Those are presented 
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i n Tab le U . 



Tlio most import.'!*’* ' information on tin- data recording lorm tor 
the purposes o£ this study was the column on diagnosis or tentative 
diagnosis. From the information recorded, t!u medical secretary was 
able to assign each patient contact to one or n.nrc of the Id categories 
in the classification system. From the cumulative totals a profile was 
generated, based on the percentage of practice in each category during 
the recording period. (See Table 5). 

Kxper imentat ion was carried out on alternate methods of 
determining a physician’s practice profile. In 15 instances the 
physician was asked on the first morning to predict his profile. Tins 
was later compared with his recorded practice profile. (See Table 6), 

A second experiment involved sending a dictating machine to the physician 
rather than having the medical secretary visit his office. One physician 
dictated patient data during the recording period and his protile was 
derived from the transcribed tapes. 

Tes t Admin i st r ation 

Based on experiments conducted early In the contract period* 
it was determined that a 100 item multiple choice test would require 
about two flours when administered by teletype. A formula was devised 
on this parameter to design an individualized test for each physician 
based or bis practice prefile. 

A conclusion was reached that it was not feasible to test 
in all 1 M categories within the two-hour* U>(> item limit set because 
the number of questions ip those categories which constituted a i ov 
percentage of the practice vcvtld be too few to hr of value. Therefore, 






the test was Revised to cover I rom h-h of the cMegorior. making up the 
gre.at rs t poro ntage of the piactice profile. In each category the six 
standard questions (two at each level of sophistication) would be assigned, 
ar.d then an additional block of five questions for each live percentage 
points of practice would bo randomly selected by the computer. In each 
block of five random questions there was one from level 1 and two each 
from levels 2 and 3. A maximum of 36 questions was allowable ji any 
one category. Table 7 is an example of how a test was* composed from 
one practice profile. 

Since the random selection of items, other than the standard 
questions, was based on the unique code number assigned to the physician, 
all participants could he assured that no two tests would be identical. 
This prohibits tost score comparisons among themselves or with any 
outside group. 

The test was administered over r portable teletype by 
telophi.ie comnmnicat f ons between the physician's office and the 
University of Wisconsin Computer Center. The items were presented 
by category, giving the physician the option of resting between 
categories. The question and the five options vert printed out at 
the teletype terminal, the physician responded by selecting one of 
the options. This brought an immediate response from the computer 
which informed him if he find answered correctly or if wrong, the 
option he should have selected. At the conclusion he received a 
summary of the results. The physician retained the teletype print- 
out for his analysis. 
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Educational Consultatio n 

An educational consultant, either from the lull-tine or clinical 
faculty of the University of Wisconsin Medical Center, was assigned to 
each participant. Selection was made in advance of profiling and testing 
with an attempt to anticipate the medical specialist who would correlate 
with the practitioner's needs. This consultant was furnished a complete 
set of the patient data with a statistical analysis of the data as veLl 
ac a computer print-out of the test and brief analysis of the results. 

He was then asked to visit the practicing physician and discuss 
Lhese with him. Utilizing the patient data* t *st results, physician's 
comments, and information 0:1 his practice setting and procedures, i\\e 
consultant and physician would jointly arrive at an educational program 
covering the next six to nine months. The educational consultant was 
asked to develop a relationship with the physician, to continue through- 
out the study period, with the Department of Vos tgradua te Medical Edu- 
cation serving as a resource and coordinator, to assist as desired. 

The consultants were encouraged to suggest a variety of 
continuing education activities to meet identified needs, e.g. appro- 
priate journal references or reprints, attendance at postgraduate courses, 
visits by faculty members to the participant's office or hospital on .1 
periodic basis, study programs at the University of Wisconsin Hospitals 
or other appropriate medical centers, etc. An honorarium of $200 was 
provided to the physician to cover expenses involved in carrying out 
the educational program, 

Evaluation 

Kd- prof i 1 j ng and re-testing were planned as a form of program 
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evaluation. Font physicians were re-profiled six months after the 
initial profile was taken. Five physicians were re-tested, two on the 
basis of the re-profiles and three on their initial profiles. 

In addition, a day-long meeting was field in May of 1 1 *69 
where participating physicians, consultants and staff members discussed 
the procedures, results and future directions of the study. 

1 J 1 , Discussion 



O btaining par tici pants 

The ease with which participants were recruited was encouraging. 
However, it must be pointed out that the 37 involved represent a biased 
s angle. They were a highly motivated group, already participating 
activeLy in continuing education activities, and considered to be secure 
in the quality of medical care they wore providing. It is significant 
that virtually all have expressed a desire to be included in any continu- 
ation of the study, and five other physicians have contacted the princi- 
pal investigator asking to he enrol loo. 

Equally encouraging. Is the fact th it the participants 
became deeply involved in the total project and were not restricted 
to providing the "laboratory" in which the study was conducted. Their 
role at the conclusion of the study period is more that of co- investi- 
gators than participants. 

Deve lopnen t of lest hank 

Limitations in the scope and structure of the test bank 
becufre apparent early in the st idy. Thirteen of the participants 
were asked to immediately review their test print-outs and designate 
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those questions which were not relevant to their practices. While this 
is not a completely objective method of determining relevance, the fact 
that their success in answering questions conouly did not seem to 
affect their judgments added to the credibility of the process. In all, 
one- third of the test questions were termed i nappropr te . In terms of 
efficiency alone, this meant that 40 minuter of each two hour test were 
essentially wasted in relation to the goals of the study* Analysis of 
the reasons for irrelevancy led to two procedural problems: 

L. classification of the test questions into 18 categories 
was not sufficiently sensitive to insure that a physician would be 
questioned on his practice profile. For example, a physician who saw 
a number of patients with allergies might find himself questioned in 
depth on metabolic problems, since they are both included in category 3. 

2. Questions arc written from an academic viewpoint, ani 
consequently the format of the question may make it inappropriate 
for testing a clinician. One example which occurred was an item 
which involved incidence of a specific condition and the options were 
in increments of five percentage points. The clinician’s response was 
that his clinical decisions would be influenced by whether the incidence 
was high or low, and the fact that he could not specify i „ within a 
few percentage points was not important. 

Co 1 lection of Practice Da ta 

The procedure and form developed for collecting practice 
data proved successful. The medical secretary found she was unable 
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to record time spent with the patient, and some physicians had a signifi 
cantly higher percentage of undiagnosed patients than others due io a 
•’■eluctancc to nrke a tentative diagnosis on the first patient visit 
before any test results were available. 1 n all other categories she 
found it possible to gather the desired data with little difficulty. 

The data confirmed that medical practices do vary greatly 
in such matters a:: patient load, method of patient contact and types 
of diseases and conditions which bring patients to the physician (See 
Tables 4, >) . 

One weakness of the procedure involves the key item in 
deve lopment of the profile, the tentative diagnosis. If it is an 
incorrect diagnosis, it is still reflected in the profile. While 
effort could be made to confirm the initial diagnosis by a later 
check of the patient pc uiI, this is not felt to bn warranted in 
terns of the potential improvement of the data. 

The major problem involved in the data collection procedure 
is the collusion bv the participants that review of a practice for 
3-4 consecutive days does not present a true practice profile. The 
only data available involves the four physicians who were re-pro* 
tiled after six months. Changes in almost all categories were less 
than five per cent, and the profiles showed little variation. Those 
changes detected involved a higher incidence of category 7, diseases 
of the respiratory system, in the November profiles and a higher 
incidence of category 1, infective arid parasitic diseases, and 
category 3, disease.* of the nervous system and sense organs, in the 
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May profiles. Si net* this i such ;> small .sample, and the participants 
Lee! quite strongly that variation docs oc'iir, effort should be iradc 
to collect sample data nvor an expanded period to determine if this 
presents a more accurate practice profile. 

The one attempt to obtain patient data by use of a dictating 
machine proved successful, and warrants further investigation since 
the expense involved in hiring and supporting a medical secretary 
for data gathering is a significant budget item. 

Physicians also demonstrated a rather consistent degree of 
accuracy in predicting their own practice profiles, within five 
pcrcertagc points in each category. The 15 who were asked to make 
advance predictions mis -estimated from two to six categories by 
more than five per cent: 

1 mis -cs t i mated two categories 

2 nis-est imated three categories 

5 nis-es t innt ed four categories 

2 nis-es t imated five categories 

5 nis-es t imated six categories 

In terns of the test composition formula, and consequently 
the effect on the study, the results would have been: 

A would have had 80 per cent the same test 

3 would have had 70 per cent the same test 

5 would have had 50 per cent the same test 

1 would have had 30 per cent the sane tost 

If, as indicated in the previous discussion, the test questions 
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and data gathering must be in sub- categories to achieve a greater sensi- 
tivity this may increase or decrease tlio physician's ability to predict 
hi s prof ile. The method shows sufficient promise in accuracy ; ml 
economy that it should be extensively explored in a ny future study. 

Test Administration 

The study demonstrated that physicians can and will be tested 
on scientific knowledge. However , problems were encountered in the 
mechanics of test admini stration. 

At times it required 3-5 hours to compLete a test designed 
for administration in two hours, due to technical problems with the 
time-shared computer. Printing of the text of the question by tele- 
type ’.-as slow and the noise of the device proved distracting to some 
participants. It is felt that the stress factors imposed by the 
teletype and computer aifectcd test performance in some cases. 

The cost of using the computer and telephone communications 
was high. Cost of admin i storing one test ranged from $50- $60 in 
computer time alone, and while telephone costs vailed depending on 
geographic location, they were substantial in rv.ny cases. 

Experimental ion was carried out in written testing. During 
one period of considerable difficulty with the computer testing 
mechanism, the staff member involved was prepared with a prir.t-out 
of the lest and if the computer malfunctioned, the test was given 
in written form. This proved to be an acceptable form of testing. 

It would, however, give the physician an opportunity to procrastinate 
in completing the test. The computer method required t'.e physician 
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to allot a specific tine period for the test, and il took precedent 
over the normal interruptions involved in nodical practice. A written 
test might not be given this same priorit) . 

Kducal i onal Cunsn Lt at i on 

There was groat variation in the volume and type of activity 
generated by the consultant-participant relationship. Kach educational 
consultant oid ..;it with the physician-participant to discuss the 
results of trie profiling and testing procedures, A wide variety of 
e ducat i ora 1 exercises was prescribed and the most successful, based 
on comments by the practitioners, were those which wo r a individually 
designed within a medical center or a teaching hospital to meet 
speci Eic needs . 

I r two instances the consultant concluded that the greatest 
assistance to the practitioner in improving hus delivery of medical 
ca*“c did not involve increased scientific Knowledge, but rather 
reorganization of office procedures. A broader application of this 
unautic i pated benefit from the study involved the insight gained by 
practitioners when they were presented data on the role the telephone 
was flaying in their contact with patients. One practitioner, as 
noted in fable 5, received an average of 5b. 2 telephone calls per 
day during the recording period. 

Analysis of the educational consultant’s role in the 
study resulted in tic conclusion taat the defects were in the 
procedure. The plan was to establish a one-to-one relationship, 
with the department playing only a supportive role when requested. 
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This assumed that the consultant would have a comprehensive knowledge of 
available resources for continuing medical education. Also, in ordei 
that the individuals could develop the desired rapport based on their 
own personalities and expertise, only general guidelines were given on 
what this relationship should be. The consultants, for the most part, 
found themselves in an uncomf or cable position due to lack of direction. 



members felt the press of other responsibilities restricted their 
available time and most participated as consultant i*o only one parti- 
cipating physician. Therefore, there was little benefit gained from 
experience. Those who did consult for more than one practitioner 
found the subsequent assignments easier and more productive, 



six members on the medical staff of a small community hospital 
participated in the study. One faculty member served as educational 
consultant for all five. In this way, the educational program could 
he devised to meet both the individual and the collective needs of 
this group of physician participants. 

Kva luat i on 

Of the five physicians who were re- tested, two on the 
basis of re trofiL.es and three on their initial profiles, four 
showed significant improvement in scoring. One showed a decrease. 
Documentation of continuing education efforts during the period 
between tests shows complete correlation; the four who improved did 
participate and the one who declined carried out no continuing 



This problem was magnified by the fact that most faculty 



experimentation involved one instance where five of the 
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rcJucnt ion as ■’ part v ’*- l ' K ' * 

V;hi k- thin wool,, a, pear to he strung -pport £o- the testing 

r ,a, anf s W and nation:,* program, a detail -aalvsK- of the -suits 

of <hc physician who under too* the most extensive National P — 

contradicts such - conclusion. His i^oveneut in test score could he 

entirely attributed to receiving see identical nations on both test, 

and answering then correctly the second time. Other factors whtch may 

have been significant were that the computer was functioning well 

dnrir.r the re-testing period and the physicians were now familiar 

wlt h the technical aspects of the testing procedure so there was less 

distraction . 
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stressed througho.il the meeting: 

• i i in .he studv , and par t i cvilar ly 
L. The procedure involved in the scuny, 

icslfr.p -.pctl-rt-. P-”*-" ““‘.otiocol to the r ». tlclwcf ■ 

t„oso rt. ware cost octi.o in „p*U« *•*« 

,„o process forced tilt- t. 

O, moot, on pro, irons. r.ci.i.f. .. •• «■*— - 

Of new learning opportunities. 

2. The rapport developed between the consultant and 
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participant, while not successful in terms of the initial study goals, 
was deemed hip, lily beneficial. The clinician gained a close associate 
to assist him in using the services and facilities of the medical 
center for the benefit of his patients and the faculty member gained 
now insights into clinical medicine which will he valuable in the day- 
to-day teaching of medical students and house staff. 

Conclus i ons 

L. physicians who are highly motivated in continuing 
ed-cation find the procedure involved in this study promising as a 
method of identifying their needs and designing appropriate educa- 
tional programs . While no conclusion can he drawn for less motivated 
physicians, the greatest value of the process would be its extension 
to those who are not now updating their medical knowledge through 
effective continuing education programs. 

2. The process cannot be used to make determinations about 
the competence of a practitioner; it deals with restricted data and 
measurement which is of use in planning continuing education. 

3, Medical practice of physicians do vary significantly, 
and it is possible to record the necessary data involved in deter- 
mining individual practice profiles. 

A broader tine base than 3-A consecutive days should 

ho investigated for gathering the data involved in a practice 
profile. 

‘i. Alternate, less expensive ncthods of deternining 
the nature of a physician’s practice (e.g., dictation of patient 
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data by lb, physician or prediction of the practice profile) show 
promise ns acceptable ways of obtaining the information required lor 

the process . 

6. Physicians can and will be tested on scientific 
knowledge, but the inadequacy of the test bank used in the study 
rendered the results invalid. 

7. Mechanisms are required to insure relevance of questions 
to clinical practice and a higher correlation between, question content 
and specific patient problems involved in the physician's practice. 

8. While a test of LOO items on the major categories of 
a practice profile appear adequate, more extensive testing would be 

dcs i rah le . 

9i Computerized testing from a remote teletype terminal 
presented problems detrimental to the objectives of the study; 
consequently it is considered advisable to utilize written testing 

in continuation of the study. 

10. while the educational consulting process did not 

provide the results expected, it performed a valuable function in 
developing rapport between the academician and the practicing 
physician which could have significant long-range benefits. 

11. Faculty members serving as consultants can assist 
the clinician in recognizing educational needs, but cannot he 
expected to have expertise on available resources and methods for 

continuing nodical education. 

1?. There is merit in working with groups of physicians 

in the same clinic or on the same hospital staff to identify common 
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ns well ns individual t i ona 1 nerds and designing educational pro- 

grams Le u'eoL both, 

13. The evaluation involving re-testing was inconclusive as 
to whether the process did identify and meet educational needs. 

Based on the reactions of all personnel involved in the 
study and analysis ol the procedures and results, it is concluded that 
the study performed under the contract proved t lx e feasibility of the 
process, a ml that it warrants continued investigation with refinement 
and changes in those portions of the study which did not provide 
conclusive results. 

This continuing study should involve experimentation with 
mechanisms which would reduce the cost of tfic process to t lie point 
that it could ho provided ou a nearly self-supporting basis. 
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PHYSICIAN INFORMATION 



Phys ic ian 
Code No. 


Physician 

Age 


Years In 
Practice 


Type Of 
Prac t ice 


Size Of 

Community 

(in 

thousands) 


N-jmbcr Of 
Phys ic ians 
In 

Community 


Medi cal 
Educat ion 




10016 


35 


9 


G.P. 


2 


3 


Wi s core' in 
1959 




13029 


46 


13 


G.P. 


2 


3 


Vis cons in 
1950 




15028 


39 


12 


G.P . 


36 


74 


Cine x nna t i 
1951 




16049 


40 


10 


G.P. 


3 


3 


Nebraska 

1956 




20021 


39 


9 


G.P. 


8 


12 


Wisconsin 

1959 




2002 5 


34 


3 


I .M . 


53 


120 


Loyola 

1960 




21069 


52 


19 


G.P. 


2 


3 


Vis cons in 
1949 




23083 


36 


10 


I.M. 


53 


120 


WIscons in 
1958 




23668 


30 


) 


G.P. 


15 


12 


Michigan 

1965 




24093 


52 


22 


G.P. 


4 


6 


Wisconsin 

1946 




25016 


34 


2 


1 .M. 


53 


120 


St . Louis 
1959 




30027 


33 


3 


G.S. 


15 


12 


Coluinb ia 
1961 




31310 


46 


14 


l’cd . 


63 


67 


Kansas 

1955 






19 



. ^oor, 


38 


7 


G.P . 


163 


A 82 


Marque tie 
1961 


36 2A7 


36 


6 


G.P. 


3 A 


cc 
! m 

I 


Georgetown 

1959 


36375 


39 


3 


G.P. 


5 


11 


Wiscons i n 
i960 


A 2 083 


6 A 


31 


G.P. 


2 


3 


Wisco.is In 
1 11 3 7 


A 3353 


33 


17 


G.r. 


i 


3 


Wisrons in 

L^j 


A 607 8 


A 8 


21 


G.P. 


8 


16 


Gli i o ago 

1 t5 A 3 


'+6 560 


♦ 3 


13 


G.P. 


~~1 ** 


1 


Nani toba 
1955 


30030 


A A 


12 


l’cd. 


35 


60 


Hali non: aim 
,.95 A 


51328 


39 


7 


G.r. 


l 


1 


Mary 1 r.nd 
I<;M 


52030 


AO 


12 


G.P. 


1 


2 


T 1 1 i noi a 
L95A 


550V A 


AO 


7 


G.P. 


13 


27 


i 

i 

i 

t 

rj 

l S 


55332 


38 


6 


G.P. 


36 


7A 


Wiscons i n 
1962 


563A3 


35 


5 


G.P. 


1 


A 


Indiana 

1*60 


6(H) A 3 


A3 


17 


G.P, 


163 


A 32 


Ginc innat i 
1951 


613V 5 


AJ 


16 


G.P. 


8 


1? 


Iowa 

1953 


6202° 


' A3 


16 


G.P. 


8 


12 


Wiscons in 
1952 


620/(8 


A3 


13 


1 .M. 


53 


120 


Nor t lives torn 
19 58 


6207 5 


A3 


12 


Pod , 


53 


120 


St. J .on it. 

1551 


62070 


31 


22 


Pod . 


53 


120 


Bnf fnlo(SUNY) 
L%5 


63053 


63 


3 A 


G.P,' 


5 


7 


Roc host or 
1032 
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Practice Profile 
Percent 


Category 


Percent 


l. 


M 


10. 




2. 


•f 


11. 


<b'S" 


3. 


LL 


12. 


/./ 


4. 


Si 


13. 


— 


5. 


i I 


14. 


ikk 


6. 


fcl 


13. 


M 


7. 


in 


16. 


M 


8. 


12 


17. 


y 


9. 




18. 


-i 



Test Composition 



Category 


Percent 


Blocks 


Selected 


Standard 


Total 


. .7. 


Jl2 


3 


x 5 = , /5^ 


<* 


2/ 


_Lt 


10-0 


JL 


X 5 = — LO- 


(e. 


lb 


5 


_isr 




X 5 = /0 


<a 


lb 


...i 


ra. 


Jl 


X 5 = _\Q_ 


(q 


Jk 


_li 


_4T 


i 


X 5 = 5" 


6 


JLL 


i 


„4_i 


i 


X 5 = 


fc 


a 


Totals 






str 




V 


(Add or 


subtract categories to 


bring total as 


near 100 as 


possible) : 




AL 


\ 




4l_ 


JLl 










TgfcL 


toa, 




Totals 



32 



